Many companies rely on computing systems and software applications to conduct their business. Computing systems and software applications deal with various aspects of companies' businesses, which can include finances, product development, human resources, customer service, management, and many other aspects. Businesses further rely on communications for a variety of purposes, such as, exchange of information, data, software, and other purposes. Computing systems/software are frequently subject to cyberattacks by viruses, malicious software or malware, and/or other ways that can be highly disruptive to operations of the computing systems/software. Malware can disrupt computer operations, gather sensitive information, and/or gain access to private computer systems. Malware is typically defined by its malicious intent and does not include any software that may cause unintentional harm due to some deficiency.
Detection of unwanted objects, such as malware, viruses, and other unwanted or untrusted objects, in an execution environment may be performed at a computer as part of endpoint detection of those unwanted objects. For example, an object, such as a file, document, program, and/or the like, may be analyzed as part of execution control to determine whether to allow execution of some portion of the object.
In one sense, shellcode is a type of malware that is embedded within an otherwise benign computer file. Computer files can be data, a program, a content file, an archive, an executable, a script, or the like. Shellcode is a piece of code typically used as a payload in the exploitation of software vulnerability. The shellcode can facilitate an unauthorized person or machine to control a computing system, causing the computer system to perform functions different from its legitimate function. Such control can be facilitated by allowing unauthorized remote access to the computer, disruption of the computing system, unauthorized transmission of data outside of the computing system, or the like. Any piece of code that is embedded within an otherwise benign-looking computer and has malicious intent can be referred to as shellcode.
An example of shellcode, polymorphic shellcode, is malware capable of disguising itself within an otherwise benign computer file. Polymorphic shellcode can be configured to mutate its internal structure in order to avoid detection. Such mutation may include analyzing the computer files and emulating the code structure of the computer files by inserting nonsense code, that resembles the computer files, within the shellcode, thus making the shellcode appear as though it is part of the computer file. The problem of reliably identifying a bounded-length mutating malware can be considered an NP-complete problem.
Some computer architectures, such as the Von Neumann computer architecture, may not be able to distinguish between instructions in the code and data in the code of a computer file. This characteristic can exploited by engineered malware to overwrite the stored control information in the file and facilitate the hijacking of the execution context of an intended process.